fix: decode answers file content explicitly as UTF-8#1970
fix: decode answers file content explicitly as UTF-8#1970sisp merged 1 commit intocopier-org:masterfrom
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #1970 +/- ##
=======================================
Coverage 98.05% 98.06%
=======================================
Files 53 53
Lines 5552 5567 +15
=======================================
+ Hits 5444 5459 +15
Misses 108 108
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
I realized that the user settings file doesn't seem to be affected by this encoding/decoding issue because it isn't shared across developer machines with potentially different operating systems unlike the answers file. Hence, when the user settings file is encoded with the system's default encoding scheme, there should be no problem, as Copier is currently using that same encoding scheme to decode the file content. |
2 participants
I've fixed a string encoding/decoding bug on Windows that broke updating a project with recorded non-ASCII character answers.
According to my investigation,
pathlib.Path.read_text()callsio.text_encoding(), and when no encoding value is provided (i.e.,encoding=None) then a default encoding is determined. When I reproduced the error on a quite clean Windows 11,sys.flags.utf8_mode == 0, soio.text_encoding(encoding=None)returns"locale"andlocale.getencoding()returns"cp1252". At the same time, theto_nice_yamlfilter passes the serialized YAML string throughto_text()without passing an explicit value to theencodingargument which defaults to"utf-8". Hence, when the answers file content is rendered as{{ _copier_answers|to_nice_yaml }}, it is always encoded as UTF-8, but on Windows 11 its content was decoded as CP-1252.The solution is to always decode the answers file content as UTF-8.
I believe we'll need to fix this problem also for loading from external data files and for loading user settings. I'll send follow-up PRs if/where needed once this one has been merged.
Fixes #1963.